Ground-Based  Sense  and  Avoid:  Enabling 
Local  Area  Integration  of  Unmanned 
Aircraft  Systems  into  the  National 
Airspace  System 

Sarah  K.  Yenson,  MIT  Lincoln  Laboratory 
Rodney  E.  Cole,  MIT  Lincoln  Laboratory 
M.  Sage  Jessee,  JHU  Applied  Physics  Laboratory 
Chris  Crowder,  SRC 
John  Innes,  US  Army 


Distribution  A:  Public  Release 

Abstract 

As  unmanned  aircraft  systems  (UAS)  become  more  important  to  the  US  military  and  other  users,  the 
pressure  to  allow  them  to  fly  in  the  national  airspace  increases.  The  greatest  impediment  to  this  is  the 
lack  of  an  alternative  means  of  compliance  with  federal  "see  and  avoid"  regulations  to  provide  the 
capability  to  avoid  airborne  conflicts  between  the  UAS  and  manned  aircraft.  To  provide  this  alternative 
means  of  compliance,  the  US  Army  is  leading  the  development  of  a  Ground-Based  Sense  and  Avoid 
System  (GBSAA).  The  system  uses  ground-based  radars,  threat  detection  and  alerting  logic,  and  decision 
support  display  aids  to  provide  an  air  picture  of  the  UAS's  operating  environment  and  follows  the  DO- 
254  and  DO-178C  standards  for  safety  critical  avionics  hardware  and  software,  respectively.  This  system 
will  allow  greater  airspace  access  and  lower  cost  operations  by  replacing  ground  observers  in  the  field 
with  a  centralized  system,  thus  consolidating  the  observer  function.  The  first  GBSAA  deployment  site  is 
expected  to  go  live  in  2016  at  Fort  Hood  Army  Air  Field,  Fort  Hood,  TX,  operating  under  the  FAA's 
Certificate  of  Authorization  process.  This  paper  provides  an  overview  of  the  system  and  of  a  human-in- 
the-loop  simulation-based  test  exercise  which  is  a  key  component  of  the  certification  of  the  system. 
During  this  test  exercise,  19  self-separation  violations  and  no  near-mid-air  collisions  (NMACs)  occurred 
during  195  hours  of  simulation,  including  many  stressing  multi-intruder  scenarios,  during  which  reported 
workload  was  low  and  situation  awareness  was  high  throughout.  All  participants  ultimately  stated  that 
the  GBSAA  system  was  appropriate  for  UAS  operations  within  the  National  Airspace  System  (NAS). 


1  Introduction 

The  Department  of  Defense  (DoD)  and  other  government  agencies,  as  well  as  universities  and 
commercial  companies,  have  a  strong  interest  in  flying  unmanned  aircraft  systems  (UAS)  in  the  National 
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Airspace  System  (NAS).  In  particular,  all  branches  of  the  US  military  have  unmet  UAS  crew  training 
needs.  Currently,  UAS  training  is  conducted  in  special  use  airspace,  but  the  UAS  must  transit  the  NAS  to 
reach  this  airspace.  In  operational  scenarios,  UAS  often  must  transit  to  Class  A  airspace  where  air  traffic 
control  (ATC)  will  provide  separation.  When  not  under  ATC  separation,  aircraft  operators  must  meet 
federal  regulations  that  require  the  operator  of  an  aircraft  to  see-and-avoid  other  aircraft.  Unmanned 
aircraft,  having  no  pilot  in  the  air  to  provide  this  function,  must  instead  provide  an  alternative  method  of 
compliance  with  these  regulations  [CFR,  2004]. 

Permission  to  operate  within  the  NAS  requires  Federal  Aviation  Administration  (FAA)  approval  of  the 
alternative  means  of  compliance  and  other  aspects  of  the  proposed  flight  operations.  Once  approval  is 
granted  by  the  FAA,  the  agency  receives  a  Certificate  of  Authorization  allowing  operators  to  perform 
specific  UAS  activities.  Today,  two  alternative  means  of  compliance  are  available:  the  use  of  a  chase 
airplane  to  accompany  the  UAS  on  its  flight,  or  a  series  of  ground  observers  in  contact  with  the  aircraft 
operator  who  can  warn  of  approaching  air  traffic.  The  Army  is  developing  a  technology-based 
alternative  means  of  complying  with  these  "see-and-avoid"  regulations  called  Ground-Based  Sense  and 
Avoid  (GBSAA).  The  development  of  this  technology  follows  a  spiral  approach  where  software  releases, 
known  as  blocks,  of  increasing  capability  are  deployed  after  extensive  testing  and  validation.  Each  new 
block  includes  increasingly  sophisticated  decision  aids  that  are  incrementally  integrated  into  the  UAS 
ground  control  station  (GCS)  and  into  the  operator's  (known  in  Army  parlance  as  the  Aircraft  Operator, 
or  AO)  workflow. 

The  GBSAA  system  replaces  the  human's  visual  acquisition  of  traffic  with  ground-based  radars  that 
sense  nearby  traffic.  The  radar  data  are  then  provided  to  operators  of  the  GBSAA  system  or,  in  future 
blocks,  directly  to  the  AO,  along  with  other  information  to  help  human  operators  maintain  the  desired 
separation  of  1  nmi  in  the  horizontal  or  1000  ft  in  the  vertical.  In  the  first  deployment  of  the  system, 
known  as  GBSAA  Block  0,  this  information  is  provided  to  a  Ground-Based  Operator  (GBO)  who,  much 
like  the  ground  observers  currently  employed,  provides  warnings  and  guidance  to  the  AO.  In  addition, 
the  system  includes  decision  support  logic  to  enable  the  GBO  (or  AO,  in  future  blocks)  to  rapidly 
prioritize  which  aircraft  represent  a  potential  future  conflict  and  to  take  action  in  a  timely  manner  to 
maintain  adequate  separation  from  all  aircraft.  Future  blocks  will  provide  explicit  maneuver  guidance. 

The  GBSAA  system  will  initially  employ  two  types  of  radars:  FAA  airport  surveillance  radars  such  as  ASR- 
9  and  ASR-11,  and  LSTAR  (Lightweight  Surveillance  and  Target  Acquisition  Radar,  SRC  Inc.)  radars.  The 
FAA  radars  provide  two-dimensional  locations  on  all  aircraft  with  sufficiently  strong  radar  returns,  and 
when  an  aircraft  provides  its  altitude  they  provide  a  full  three-dimensional  location  estimate.  The  LSTAR 
radars  provide  three-dimensional  position  data  on  all  aircraft  with  sufficient  radar  returns.  The  data 
from  these  radars  is  fused  using  a  Kalman  filter  tracker  to  improve  track  accuracy  and  the  fused  tracks 
are  passed  through  a  track  classifier  to  remove  false  tracks  such  as  those  caused  by  birds.  Figure  1  shows 
the  GBSAA  Block  0  system  architecture. 

Once  a  clean  air  picture  is  produced,  each  aircraft  is  evaluated  by  the  Assess  algorithm  to  determine  if  it 
poses  a  threat  to  the  UAS  and  if  so,  its  relative  priority  among  all  threat  aircraft.  If  an  aircraft  is  assessed 
as  a  threat,  a  prioritized  alert  is  issued.  There  are  two  levels  of  alerts:  proximate  and  warning.  A 
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proximate  alert  lets  the  GBO  know  there  is  an  aircraft  that  must  be  watched,  and  the  operator  may  or 
may  not  take  action.  A  threat  alert,  in  contrast,  is  provided  when  a  developing  situation  necessitates 
operator  action.  In  addition,  the  Assess  algorithm  makes  a  maneuver  recommendation  based  on  the 
behavior  of  current  intruders.  Initially  this  recommendation  will  not  be  displayed  to  the  operator  and 
will  only  be  recorded  for  post-mission  analysis.  Once  the  maneuver  logic  is  sufficiently  robust  it  will  be 
enabled  as  a  display  product  in  future  blocks. 
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Figure  1:  GBSAA  Block  0  system  architecture 

Lastly,  the  GBSAA  system  is  a  dual-display  system.  A  traffic  display  shows  the  overall  air  picture  and 
basic  information  on  intruder  prioritization  and  system  status.  An  alert  display  provides  a  streamlined 
air  picture  of  only  traffic;  it  also  includes  more  detailed  information  about  the  alerting  intruders. 

The  Army  is  in  the  final  stages  of  transitioning  the  GBSAA  system  from  development  to  production  and 
deployment.  The  software  is  written  to  DO-178C  standards,  the  hardware  meets  DO-254  standards,  and 
the  Army  Engineering  Directorate  is  providing  the  airworthiness  certification.  As  part  of  the  certification 
process,  the  Army  requires  a  safety  analysis  of  the  full  GBSAA  system.  This  safety  case  includes,  but  is 
not  limited  to,  analyses  of  the  GBSAA  subsystems,  potential  failure  modes,  risk  ratios,  and  requirements. 
It  also  requires  a  demonstration  of  the  usability  of  the  GBSAA  system  by  the  UAS  operators;  in  this  case, 
the  AO  and  GBO  working  together  as  a  team.  An  important  component  of  this  requirement  was  a 
human-in-the-loop  (HITL)  laboratory  simulation  called  the  Operational  Effectiveness  (OE)  evaluation  and 
is  the  focus  of  this  discussion.  In  this  simulation  the  production  code  for  the  Assess  algorithm  and  the 
Traffic  and  Alert  displays  were  run  on  production  hardware;  synthetic  intruder  aircraft  tracks  were 
injected  into  the  testbed  system  in  which  the  UAS  was  simulated  and  the  GCS  controls  emulated. 
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Although  the  OE  evaluation  is  included  as  part  of  the  Army's  safety  case  for  the  GBSAA  system,  a 
discussion  of  the  majority  of  the  safety  case  is  outside  of  the  scope  of  this  work.  Additionally,  although 
data  informing  the  safety  case's  risk  ratio  analysis  were  collected  during  this  evaluation,  the  analysis  was 
performed  by  US  Army  analysts  and  is  not  included  herein. 

The  GBSAA  OE  evaluation  for  Block  0  took  place  at  the  Massachusetts  Institute  of  Technology  -  Lincoln 
Laboratory  (MIT  LL)  from  22  March  through  10  April  2015.  This  evaluation  focused  on  the  GBSAA  GBO's 
ability  to  successfully  utilize  the  GBSAA  system  as  measured  by  user  surveys  and  by  an  analysis  of  the 
ability  of  GBOs  to  maintain  separation  from  other  air  traffic.  The  OE  test  was  conducted  as  a  HITL 
evaluation  that  examined  GBO  utilization  of  the  system  across  three  different  scenario  types: 

1.  Use  Case:  14  synthetic  flights  representative  of  end-to-end  missions  based  at  Robert  Gray  Army 
Airfield  (KGRK)  at  Fort  Hood,  TX  (each  20  -  25  minutes  in  duration). 

2.  Envelope  Case:  125  very  challenging  traffic  encounters  involving  multiple  intruders  in 
unstructured  airspace  (each  4-5  minutes  in  duration)  designed  to  help  define  the  outer 
performance  envelope  of  GBSAA  operations. 

3.  Risk  Ratio:  66  short,  multi-intruder  vignettes  designed  to  support  the  human  effectiveness 
component  of  the  safety  case  for  the  GBSAA  system. 

In  all  scenarios,  ownship  performance  and  speeds  were  based  on  those  of  a  Gray  Eagle  aircraft.  Intruder 
speeds  and  altitudes  were  representative  of  those  of  helicopters  and  general  aviation  aircraft. 

A  total  of  eight  individuals  with  pilot  and/or  ATC  backgrounds  participated  in  the  OE  evaluation  as  GBOs. 
All  selected  GBOs  were  familiar  with  the  system's  basic  concept  of  operations  and  functionality.  The  test 
environment  was  established  to  replicate  a  GBO  workstation  that  will  be  used  at  each  fielded  site. 
Surrogate  AOs  were  included  in  the  use  case  and  risk  ratio  cases  to  simulate  the  crew  coordination 
required  under  normal  operating  constraints.  In  preparation  for  runs  for  record,  GBOs  participated  in  a 
two-hour  training  period,  designed  as  a  refresher  course  for  the  display  symbology  definitions  and  test- 
specific  standard  operating  procedures.  Afterwards,  the  GBOs  were  provided  approximately  two  hours 
of  hands-on  simulation  time  in  order  to  familiarize  themselves  with  the  GBSAA  displays  and  test 
procedures  conducted  throughout  the  runs  for  record. 

The  OE  evaluation  was  designed  to  verify  that  the  GBSAA  system  meets  the  human  performance 
requirement  outlined  in  the  GBSAA  System  Level  Requirement  SRP2B0-340.  This  requirement  states 
that  the  GBSAA  system  controls  and  displays  shall  be  arranged  and  located  so  they  are  functionally 
visible.  Additionally,  this  evaluation  intended  to  assess  that  the  GBSAA  system  is  safe,  usable,  and  useful 
from  the  perspective  of  the  intended  user  and  to  evaluate  if  the  system  is  capable  of  safely  operating 
within  its  intended  area  of  operation.  The  final  goal  of  OE  testing  was  to  capture  HITL  data  necessary  to 
calculate  the  GBO  logic  ratio  as  part  of  the  GBSAA  safety  case.  Similar  human  performance  evaluations 
were  conducted  for  the  Traffic  Alert  and  Collision  Avoidance  System  (TCAS)  [Chappell,  1988]  and  for  a 
flight  data  management  tool  for  tower  air  traffic  controllers  [Reynolds  et  al.,  2013]. 

Five  key  organizations  participated  in  the  development,  integration,  and  runs  for  record  (RFR)  for  the  OE 
test  event.  The  Army  sponsored  the  OE  evaluation  and  is  the  system  integrator.  MIT  LL  and  Johns 
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Hopkins  University  Advanced  Physics  Laboratory  shared  responsibility  for  planning  and  executing  the  OE 
evaluation.  SRC,  Ensco,  and  Kutta  Technologies  participated  in  system  architecture,  subsystem,  and 
code  development  and  testing. 


2  OE  Evaluation  Design  and  Method 

2.1  Test  Needs  and  Purpose 

To  evaluate  the  OE  test  goals,  three  types  of  tests  were  used:  envelope  tests,  use  case  tests,  and  risk 
ratio  tests.  The  envelope  tests  were  designed  as  stressing  cases,  while  the  use  case  tests  were  designed 
to  verify  that  the  system  is  usable  in  simulated  scenarios  similar  to  the  context  in  which  the  system  will 
be  deployed.  Finally,  risk  ratio  tests  were  conducted  to  capture  critical  data  needed  for  the  safety  case. 
Further  details  of  each  test  are  provided  below. 

2.1.1  Envelope  Test  Scenarios 

The  envelope  tests  were  designed  to  stress  the  human-inclusive  system's  ability  to  maintain  self¬ 
separation  in  an  unstructured  airspace  with  multiple  unpredictable  intruders  in  the  vicinity  of  the 
ownship.  This  was  accomplished  by  presenting  a  GBO  with  a  series  of  trials  that  lasted  four  to  five 
minutes  each.  Each  trial  began  with  up  to  four  potential  intruders  located  within  ten  nmi  of  an 
eastbound  ownship.  As  the  trial  progressed,  the  intruder  aircraft  flew  predefined  flight  paths  in  the 
vicinity  of  the  ownship.  These  flight  paths  were  designed  to  produce  proximate  or  warning  alerts  that 
would  motivate  the  GBO  to  make  a  maneuver  decision.  The  GBO  initiated  each  maneuver  decision  by 
using  a  command  interface  shown  in  Figure  2. 


Figure  2:  GBO/AO  command  input  widget 

Of  the  420  total  intruders  seen  throughout  the  envelope  tests,  209  were  designed  to  produce  a 
proximate  alert,  while  211  were  designed  to  produce  warning  alerts.  However,  it  is  important  to  note 
that  the  expected  alert  status  of  each  pre-defined  intruder  track  may  not  have  consistently  developed 
into  an  alert  because  of  the  variability  in  GBO  maneuver  responses.  In  addition  to  the  alert  status  of 
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each  intruder,  the  test  team  designed  the  intruders  so  that  they  captured  the  full  range  of  the  following 
parameters. 

•  Intruder  count  -  One,  two,  three,  or  four  intruders 

•  Intruder  approach  geometry  relative  to  ownship  -  Head  on,  overtaking,  90  degree  intrusion, 
front  oblique,  or  back  oblique 

•  Speed  relative  to  ownship  -  Greater  than,  less  than,  or  the  same 

•  Altitude  relative  to  ownship— Greater  than,  less  than,  or  the  same 

•  Acceleration  parameters  -  Accelerating,  decelerating,  or  none 

•  Altitude  trend  -  Climbing,  descending,  or  level 

•  Predictability  -  Heading  change  of  intruder,  or  no  heading  change  of  intruder 

The  closest  point  of  approach  (CPA)  was  then  evaluated  for  each  intruder  across  each  trial  in  order  to 
determine  if  the  GBO  was  able  to  maintain  self-separation  throughout  the  very  difficult,  if  not  worst- 
case,  situations.  In  addition  to  CPA,  workload  and  situational  awareness  ratings  were  captured  at  the 
end  of  each  trial  in  order  to  determine  relationships  between  performance,  intruder  characteristics,  and 
subjective  assessments  under  varying  conditions. 


Figure  3  shows  an  example  of  an  envelope  scenario.  In  this  scenario  there  are  two  westbound  head-on 
intruders,  one  southeast-bound  oblique  intruder,  and  one  northeast-bound  90-degree  intruder.  All 
intruders  are  shown  in  green,  while  the  ownship  is  shown  in  blue.  For  all  scenarios,  the  ownship 
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maintained  a  constant  speed  and  altitude  throughout  the  scenario.  These  speeds  ranged  from  85  to  110 
kn  and  altitudes  ranged  from  3000  to  10,000'.  The  intruder  altitudes  varied  through  the  same  altitude 
range,  with  speeds  ranging  from  90  to  150  kn.  Intruder  turn  rates  varied  between  two  and  3.5  degrees 
per  second  and  vertical  rates  varied  between  300  and  1000  fpm. 

2.1.2  Use  Case  Test  Scenarios 

In  contrast  to  the  envelope  tests,  the  use  case  tests  were  designed  to  mimic  a  real-life  mission  for  an 
unmanned  aircraft  operating  out  of  Robert  Gray  Army  Air  Field  (KGRK)  at  Fort  Hood,  Texas.  These 
scenarios,  lasting  20  to  25  minutes  each,  began  with  a  takeoff  from  KGRK.  The  ownship  then  proceeded 
to  the  restricted  area  near  Fort  Hood  (R-6302)  via  a  prescribed  corridor.  Once  in  the  restricted  area,  the 
ownship  performed  some  simple  maneuvers  in  the  restricted  area,  followed  by  a  return  to  KGRK  via  the 
corridor.  Throughout  the  scenario,  the  operator  identified  potential  traffic  threats  and  communicated 
maneuver  recommendations  to  a  confederate  AO  located  in  a  separate  room,  who  would  then  input  the 
maneuver  via  the  same  command  interface  used  in  the  envelope  tests.  Communication  with  the  AO 
was  achieved  through  an  open  telephone  line;  the  GBO  and  AO  both  wore  headsets  to  reduce 
environmental  distraction.  The  GBO's  communications  were  recorded  using  a  lapel  microphone  to 
facilitate  an  analysis  of  communication  latency. 

Fourteen  use  case  scenarios  were  used  in  the  OE  evaluation.  Traffic  levels  and  patterns  seen  in  these 
scenarios  were  based  on  an  airspace  analysis  of  the  Fort  Hood  region  and  included  military  helicopters 
and  civilian  fixed-wing  aircraft  operating  outside  the  restricted  area,  and  military  aircraft  operating 
within  the  restricted  area.  Four  missions  had  low  density  traffic,  with  the  number  of  intruders  ranging 
from  two  to  four  intruders.  The  remaining  ten  missions  were  split  between  medium  density  traffic,  with 
eight  to  ten  intruders,  and  high  density  traffic,  with  16  to  20  intruders.  Figure  4  illustrates  an  example  of 
a  low  density  use  case  scenario.  Note  that  the  track  shown  in  blue  indicates  the  ownship  track.  The 
remaining  tracks  are  all  intruder  tracks. 
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Figure  4:  Example  of  low-density  use  case  scenario 

In  designing  the  intruders'  flight  paths,  various  Fort  Hood  flight  rules  and  procedures  informed  traffic 
placement  and  timing.  For  example,  at  KGRK  manned  aircraft  fly  on  the  east  side  of  the  runway,  while 
unmanned  aircraft  operate  on  the  west  side.  Additionally,  military  helicopters  are  authorized  to  fly  at 
specific  altitudes  in  a  corridor  between  KGRK  and  R-6302. 

With  these  rules  in  mind,  a  collection  of  intruder  flight  paths  and  a  collection  of  ownship  mission  paths 
were  created  and  used  to  develop  the  14  missions.  For  each  mission,  these  common  flight  paths  were 
modified  for  location,  timing,  altitude,  and  speed  to  facilitate  reasonable  potential  encounters 
throughout  the  mission.  However,  it  must  be  noted  that  the  encounters  assumed  no  deconfliction 
efforts  from  either  ATC  or  the  intruder  pilot,  both  of  whom  would  typically  prevent  such  encounters  - 
for  example,  operators  noted  that  ATC  would  never  allow  another  aircraft  to  take  off  from  GRK  when  an 
unmanned  aircraft  was  inside  the  transit  corridor  returning  to  the  airfield. 

Unlike  the  envelope  tests,  test  parameters  other  than  traffic  density  were  not  controlled,  since  a  main 
goal  of  the  use  case  tests  was  to  imitate  a  realistic  mission  to  the  extent  possible  within  the  confines  of 
an  experimental  testbed.  To  accomplish  this  goal  of  reasonable  realism,  the  use  case  scenarios  were 
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reviewed  by  an  ATC  subject  matter  expert  at  MIT  LL  and  were  presented  to  subject  matter  experts 
associated  with  the  Army  for  review. 

As  for  the  envelope  tests,  for  each  trial  the  CPA  was  evaluated  to  determine  if  self-separation  was 
maintained,  and  workload  and  situation  awareness  ratings  were  collected. 

2.1.3  Risk  Ratio  Test  Scenarios 

The  risk  ratio  tests  were  designed  to  test  GBO  responses  to  intruder  events  that  occur  in  unstructured 
airspace.  As  compared  to  the  envelope  tests,  risk  ratio  trials  had  slightly  fewer  intruders,  but  were  still 
highly  stressing  and  in  some  cases  beyond  what  would  normally  be  expected  in  typical  operations  in  the 
NAS.  As  in  the  use  cases,  the  risk  ratio  tests  required  communication  between  the  GBO  and  AO  for 
recommended  maneuvers.  These  data  were  collected  in  support  of  the  GBO  logic  ratio  calculation  that 
is  included  in  the  GBSAA  safety  case  documentation. 

In  designing  the  risk  ratio  scenarios,  several  criteria  were  taken  into  account.  First,  the  encounters  were 
designed  with  a  primary  intruder  that  would  trigger  a  threat  alert  if  no  action  was  taken.  Though 
additional  intruders  were  present  and  potentially  causing  alerts,  these  intruders  were  secondary  targets. 
Additionally,  the  encounters  occurred  over  a  range  of  angles,  similar  to  the  approach  geometry 
classifications  used  in  the  envelope  scenarios.  The  encounters  also  were  designed  using  standard  rate 
turns  and  approximately  500  fpm  altitude  changes.  Finally,  the  scenarios  were  designed  so  that  with  no 
GBO  intervention  the  CPA  would  fall  within  0.5  to  1  nmi  horizontally  and  less  than  1000'  vertically. 

Many  of  the  envelope  scenarios  already  met  these  criteria,  so  they  were  reviewed  and  modified  by  a 
subject  matter  expert  for  reuse  as  risk  ratio  scenarios.  The  modifications  primarily  involved  adjustments 
to  the  ownship  and  intruders'  paths  and  altitudes  in  order  to  better  fulfill  the  specified  criteria.  Of  the 
65  risk  ratio  scenarios,  19  scenarios  were  unique  to  the  risk  ratio  scenarios  while  the  remaining  46  were 
modified  envelope  scenarios.  Figure  5  provides  an  example  of  a  risk  ratio  scenario.  As  the  majority  of 
these  scenarios  were  repurposed  from  existing  envelope  scenarios,  the  intruder  and  ownship 
performance  characteristics  are  the  same  as  those  for  the  envelope  scenarios. 
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3  GBO  Selection  and  Schedule 
3.1  GBO  Qualifications 

Eight  GBOs  were  selected  from  a  consortium  of  government  and  contractor  organizations  from  the  ATC, 
pilot,  and  UAS  operator  communities.  The  selected  GBOs  were  required  to  meet  minimum  qualifications 
needed  to  perform  the  GBO  functions.  These  minimum  requirements  included  500  hours  as  pilot-in¬ 
command  of  manned  aircraft  or  one  year  as  a  facility-rated  air  traffic  controller.  All  selected  GBOs  were 
familiar  with  the  system's  concept  of  operations  and  functionality.  GBOs  were  required  to  travel  to  MIT 
LL  for  a  one-week  period  from  either  March  29th  to  April  3rd  or  April  5th  to  April  10th,  2015.  In  lieu  of  the 
80-hour  qualification  training  required  for  qualification  as  an  official  GBO,  GBO  operators  went  through 
an  abbreviated  GBSAA  GBO  familiarization  session  that  included  a  classroom  training  session  and  a 
hands-on  simulation  session. 

Five  of  the  GBOs  had  ATC  experience  only,  while  the  remaining  three  were  pilots.  Of  the  three  pilots, 
one  GBO  had  experience  as  a  pilot  and  air  traffic  controller  and  one  GBO  had  experience  as  a  small  UAV 
operator  and  pilot.  The  controllers  reported  experience  ranging  from  three  to  20  years,  with  the 
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average  being  13.8  years,  and  the  pilots'  reported  flight  time  ranged  from  2,500  to  4,500  flight  hours, 
with  an  average  of  3,167  flight  hours. 

Most  operators  reported  familiarity  with  the  GBSAA  concept  and  a  general  working  knowledge  of  the 
flight  rules  and  airspace  in  and  around  Fort  Hood.  Operator  familiarity  with  the  GBSAA  concept 
averaged  4.4  on  a  5-point  scale,  where  5  corresponded  to  "very  familiar";  the  lowest  rating  was  a  1  and 
highest  was  a  5.  They  were  similarly  familiar  with  the  Fort  Hood  flight  procedures,  averaging  4.1  on  the 
same  5-point  scale;  the  lowest  score  was  a  3  and  highest  was  5. 

3.2  Operational  Effectiveness  GBO  Training  Session 

There  were  two  primary  components  of  the  GBO  training  session.  First,  the  previously  mentioned 
classroom  portion  provided  an  overview  of  the  GBSAA  displays,  test  procedures,  and  other  test 
information.  This  briefing  covered  logistics  associated  with  test  conduct,  differences  in  the  GBO  input 
between  different  test  types,  guidelines  for  interpreting  the  assessment  algorithm  alerts,  a  review  of  the 
operating  area  characteristics  associated  with  each  test  type,  and  phraseology  guidance  for  traffic 
communications  between  the  GBO  and  AO.  GBOs  received  the  following  guidance  on  how  to  respond  to 
the  alerts. 

•  Proximate  alert:  GBO  may  use  discretion  in  providing  traffic  information  and/or  maneuver 
information  to  AO 

•  Warning  alert:  GBO  must  provide  traffic  information  and  maneuver  information  to  AO  if  outside 
the  3.4  nmi  area  surrounding  the  airport,  known  as  the  Terminal  Area  Alert  Zone  (TAAZ) 

One  of  the  key  areas  within  this  two-hour  training  session  were  the  voice  communication  procedures 
used  to  recommend  maneuvers  to  a  confederate  acting  as  an  air  vehicle  operator.  In  order  to  optimize 
crew  coordination,  the  following  phraseology  guidance  was  provided  for  maneuver  recommendations. 

•  Recommend  <right/left>  turn  to  <heading>.  Traffic  is  <bearing>,  <range>,  <course>,  <altitude>, 
<trend> 

o  Example:  "Recommend  right  turn  to  heading  030  degrees,  intruder  2  o'clock,  2.2  miles, 
eastbound,  4,300  ft,  170  knots,  level" 

In  addition  to  the  briefing,  the  GBOs  received  an  operator  packet  that  included  an  overview  of  the 
display  symbology,  criteria  for  maintaining  self-separation,  and  phraseology  examples.  The  packet  also 
provided  maps  with  an  overview  of  the  operating  area  for  the  use  case  test,  including  manned  and 
unmanned  corridors  within  the  operational  area.  This  operator  packet  could  be  used  by  the  GBOs  at  any 
time  during  the  runs  for  record. 

Once  the  classroom  training  session  was  complete,  GBOs  began  the  second  portion  of  training  in  which 
they  utilized  the  GBSAA  system  in  the  simulated  environment.  During  this  two-hour  period,  GBOs 
completed  training  trials  for  each  of  the  test  types.  This  allowed  them  to  gain  hands-on  experience  with 
the  GBSAA  system  and  the  specifics  of  how  it  would  be  utilized  in  each  test  type.  For  example,  during 
the  risk  ratio  and  use  case  trials  the  GBOs  focused  on  applying  their  knowledge  of  the  system  symbology 
and  algorithms  for  conducting  operationally  relevant  procedures  with  an  AO.  In  the  envelope  test 
training  the  GBOs  provided  their  own  maneuver  inputs  to  avoid  oncoming  traffic.  Here  it  was  important 
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for  GBOs  to  familiarize  themselves  with  the  maneuver  input  software.  In  all  cases  the  GBOs  gained 
familiarity  with  the  pace  and  structure  of  each  test  type. 

3.3  Operational  Effectiveness  Daily  Test  Schedule 

Run  for  Record  test  days  began  at  approximately  7:30  am  with  a  quick  overview  of  the  day's  activities 
and  test  station  and  test  block  assignments.  Participants  then  proceeded  to  the  test  lab  for  runs  for 
record.  On  an  average  day,  they  completed  four  to  five  test  blocks  before  breaking  for  an  hour-long 
lunch.  Operators  had  the  option  of  taking  a  10-minute  break  between  each  block. 

After  the  lunch  break,  operators  received  new  test  station  and  test  block  assignments.  They  completed 
another  four  to  five  test  blocks,  again  with  discretionary  10-minute  breaks  between  each  block.  At  the 
end  of  the  day's  runs,  a  short  after-action  review  was  conducted,  where  the  day's  activities  were 
reviewed,  any  issues  with  the  system  were  noted,  and  operators  provided  brief  feedback  on  the  day's 
activities.  Operators  were  dismissed  between  5  and  6  pm. 


4  Simulation  Test  Environment 

The  simulation  test  environment  was  located  in  MIT  LL's  Air  Traffic  Management  (ATM)  and  Ground 
Control  Station  (GCS)  labs.  Figure  6  illustrates  the  physical  layout  for  the  two  labs. 
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Figure  6:  GBSAA  OE  lab  layout  and  test  stations 

The  GCS  lab,  located  on  the  left,  housed  the  two  envelope  test  stations.  It  also  contained  the  two  AO 
stations  and  the  supporting  hardware  necessary  to  run  the  OE  testbed.  The  test  stations  used  for  the 
use  case  and  risk  ratio  scenarios  were  located  in  the  ATM  lab,  shown  on  the  right  side  of  Figure  6.  This 
physical  separation  was  designed  to  simulate  the  distributed  nature  of  the  GBO  and  AO  coordination 
that  would  be  observed  in  situ,  which  requires  voice  communications  over  a  headset.  The  use  case 
stations  were  converted  to  risk  ratio  stations  after  all  use  cases  were  complete.  These  test  stations  were 
utilized  for  the  risk  ratio  test  instead  of  the  envelope  stations  because  of  the  GBO  to  AO  coordination 
requirement. 
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During  the  envelope  trials  the  GBOs  were  required  to  wear  headsets  in  order  to  ensure  that  audible 
alerts  were  presented  in  a  similar  manner  to  a  live  situation.  This  also  ensured  that  ambient  noise  did 
not  serve  as  a  significant  distractor. 


Figure  7:  Alert  Display  (left)  and  Traffic  Display  (right) 

For  all  three  test  types,  participants  used  the  Alert  Display,  located  on  the  left  side  of  the  workstation, 
and  the  Traffic  Display,  located  in  the  center  of  the  workstation.  The  right-hand  monitor  provided  the 
scenario  start-up  and  delivery  graphical  user  interface,  and  in  the  case  of  the  envelope  tests,  the 
heading  command  widget  shown  in  Figure  lError!  Reference  source  not  found..  The  Alert  and  Traffic 
Displays  are  shown  in  Figure  7. 

In  this  study,  since  only  the  Assess  algorithm  and  the  Traffic  and  Alert  Displays  were  evaluated,  MIT  LL 
developed  software  designed  to  facilitate  scenario  delivery,  data  recording,  and  data  emulation  so  that  a 
realistic  simulation  of  the  GBSAA  system  could  be  evaluated.  Figure  8  details  the  system  architecture 
used  for  the  OE  evaluation  and  how  the  data  is  supplied,  shared,  and  recorded  in  the  system  testbed. 
Tools  developed  for  this  evaluation  include  software  to  deliver  scenarios  and  surveys  to  the  operator. 
Running  these  delivery  tools  initiated  data  recording  tools  that  record  ownship  and  intruder  position, 
altitude,  and  alert  data.  Additional  software  was  written  to  emulate  GBSAA  health  and  status,  and  a 
final  tool  was  developed  to  monitor  the  status  of  the  OE  tool  suite  described  here. 

In  Figure  8,  red  blocks  represent  production  software.  Blue  blocks  denote  software  tools  developed  for 
the  OE  testbed  system,  including  tools  designed  for  data  emulation  and  recording.  Software  tools 
written  to  enable  scenario  delivery  and  operator  control  of  the  ownship  for  the  OE  evaluation  are  shown 
in  orange,  and  green  indicates  tools  that  were  developed  to  enable  the  development  of  the  synthetic 
scenarios  used  during  OE  testing. 
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Figure  8:  OE  technical  system  architecture 


5  Operational  Effectiveness  Test  Results 

5.1  Results 

Two  key  criteria  used  in  the  separation  analysis  were  the  TCAS  definition  of  a  near-mid-air  collision 

(NMAC)  and  the  self-separation  standard  designed  into  the  GBSAA  system. 

•  NMAC  -  As  defined  by  TCAS  standards,  an  NMAC  is  when  separation  between  the  ownship  and 
an  intruder  aircraft  is  less  than  100ft'  vertically  and  500ft'  horizontally  [RTCA,  2013]. 

•  Self-separation  -  For  the  GBSAA  system,  a  self-separation  violation  is  defined  as  when 
separation  between  the  ownship  and  an  intruder  aircraft  is  less  than  1000ft'  vertically  and  1  nmi 
horizontally.  This  definition  was  developed  prior  to  the  existence  of  the  emerging  RTCA  well- 
clear  definition  and  is  based  on  the  Army's  investigation  of  several  self-separation  criteria 
[Hendrickson,  2015a,  2015b]. 

Overall,  the  operators  and  the  GBSAA  system  performed  well  during  the  OE  evaluation.  The  two  week 

RFR  resulted  in  the  1,608  trials  for  use  in  the  separation  analysis,  with  5,856  intruders  encountered. 

Across  these  encounters,  no  NMACs  and  19  self-separation  violations  were  recorded. 

5.1.1  Total  Trials  and  Intruders 

During  the  runs  for  record  testing,  a  total  of  1,640  trials  of  205  unique  scenarios  were  completed,  with 

the  eight  GBOs  encountering  5,960  intruders.  However,  during  the  test  conduct  four  scenarios  were 
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identified  by  the  GBOs  and  other  test  personnel  as  being  inappropriate  for  use  in  assessing  the  usability 
of  the  GBSAA  system  by  the  GBOs,  and  were  marked  for  closer  review.  These  trials  each  utilized  the 
same  core  intruder  set— namely  two  overtaking  intruders  rapidly  accelerating  from  70  knots  up  to  140 
knots  with  one  of  them  turning  directly  towards  the  ownship.  These  intruders  accelerated  very  quickly 
towards  the  ownship  in  a  way  that  was  specifically  disadvantageous  to  maintaining  separation.  Test 
personnel  and  GBOs  agreed  that  when  faced  with  these  intruders,  successful  maintenance  of  the  self¬ 
separation  standard  would  be  beyond  the  capability  of  any  SAA  system.  It  is  important  to  note  that  an 
SAA  system  itself  cannot  necessarily  avoid  more  maneuverable  and  potentially  hostile  intruders  that  can 
in  essence  hunt  down  the  UAS. 

These  trials— four  envelope  and  one  risk  ratio— were  subsequently  eliminated  from  the  test  data  and 
subsequent  analysis.  After  removing  these  four  scenarios,  the  data  set  for  analysis  consists  of  201 
unique  scenarios,  1608  trials,  and  5,856  intruders.  All  reported  results  in  this  report  use  this  dataset 
which  is  summarized  in  Table  1.  Additionally,  the  total  number  of  test  blocks  and  trials  completed  are 
presented  to  illustrate  the  breadth  of  the  human  in  the  loop  data  that  were  collected.  Here,  a  test  block 
is  defined  as  a  set  of  trials  that  usually  lasted  60  minutes.  The  envelope  test  had  a  total  of  12  blocks 
consisting  of  10  or  11  trials  depending  on  the  specific  block.  The  use  case  test  had  a  total  of  seven 
blocks,  each  with  2  trials,  while  the  risk  ratio  test  had  6  blocks,  each  with  11  distinct  trials.  Note  that 
each  block  did  not  contain  the  same  number  of  trials  across  each  test  type.  For  example,  the  use  case 
blocks  were  fewer  in  number  but  greater  in  duration  compared  to  the  envelope  or  risk  ratio  trials.  Table 
2  provides  a  breakdown  of  the  total  trials,  blocks,  and  intruder  encounter  frequency  for  each  of  the  test 
types. 


Overall  OE  Intruder/Trial  Count 

Total  #  of  Intruders 

5856 

Blocks  Completed 

200 

Trials  Completed 

1608 

Table  1:  Total  trials,  blocks  and  intruder  encounter  frequency 


Envelope  Test  Status 

Use  Case  Test  Status 

Risk  Ratio  Status 

Total  #  of  Intruders 

3272 

Total  #  of  Intruders 

1176 

Total  #  of  Intruders 

1408 

Blocks  Completed 

96 

Blocks  Completed 

56 

Blocks  Completed 

48 

Trials  Completed 

976 

Trials  Completed 

112 

Trials  Completed 

520 

Table  2:  Total  trials,  blocks,  and  intruder  encounter  frequency  per  scenario  type 


5. 1.1.1  Total  Number  of  Intruders 

Of  the  5,856  intruders,  4,354  of  them  produced  at  least  one  alert.  These  intruders  break  down  as 
follows: 

•  Envelope  trials:  2,463  alerting  intruders 

•  Use  case  trials:  844  alerting  intruders 

•  Risk  ratio  trials:  1,047  alerting  intruders 
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5.1.2  Total  Number  of  Self-Separation  Violations 

Self-separation  violations  were  defined  as  when  an  intruder  is  closer  than  1  nmi  laterally  and  1000' 
vertically  to  the  ownship.  Intruders  that  are  1  nmi  or  farther  laterally,  or  1000'  or  more  vertically  from 
the  ownship  will  not  incur  a  self-separation  violation.  Both  conditions  must  be  met  to  result  in  a  self¬ 
separation  violation,  so  an  intruder  1  nmi  laterally  and  999'  vertically  from  the  ownship  will  not  result  in 
a  violation,  while  an  intruder  0.99  nmi  laterally  and  999'  vertically  from  the  ownship  will. 


Out  of  1,608  trials  and  5,856  intruders,  a  total  of  19  self-separation  violations  occurred.  The  19 
violations  were  spread  across  18  trials,  with  one  use  case  experiencing  two  self-separation  violations  in  a 
single  trial.  Error!  Reference  source  not  found.  The  small  fraction  of  intruders  and  alerts,  including  the 
stressing  cases  in  the  envelope  test,  resulting  in  separation  violations  provides  evidence  that  the  GBSAA 
system  is  capable  of  safely  providing  the  sense-and-avoid  function  for  unmanned  aircraft.  This  is  further 
reinforced  in  the  NMAC  analysis  below. 


5.1.2  Total  Number  of  NMACs 

A  near-mid-air  collision  is  defined  as  when  an  intruder  is  closer  than  500'  laterally  and  100'  vertically  to 
the  ownship.  As  in  the  definition  for  self-separation,  intruders  that  are  500'  or  farther  laterally  or  100' 
or  more  vertically  from  the  ownship  will  not  incur  a  NMAC  violation. 


Over  the  duration  of  the  OE  evaluation,  no  NMACs  occurred  in  any  of  the  test  types.  The  lack  of  any 
NMACs  across  1,608  trials  and  5,856  intruders,  including  the  stressing  cases  in  the  envelope  test, 
provides  evidence  that  the  GBSAA  system  is  capable  of  safely  providing  the  sense-and-avoid  function  for 
unmanned  aircraft. 


5.1.4  Minimum  Separation  Distances 

In  addition  to  quantifying  the  number  of  NMACs  and  self-separation  violations,  the  OE  evaluation 
analysis  looked  at  the  CPA  for  each  trial.  For  this  analysis,  a  weighted  slant  range  for  each  intruder 
report  was  calculated  by  R  =  A fh 2  +  (5 z)2  ,  where  h  =  horizontal  range  and  z  =  vertical  range;  in  each 
trial,  the  minimum  value  for  R  was  then  used  as  that  trial's  CPA.  This  weighted  slant  range  provides 
normalized  input  from  both  the  horizontal  and  vertical  miss  distances  in  a  ratio  equal  to  that  of  an 
NMAC  (500'  horizontaklOO'  vertical)  and  accounts  for  the  difference  in  the  level  of  safety  provided  by 
vertical  vs  horizontal  separation  distances.  For  all  trial  types,  the  average  minimum  slant  range 
separation  for  each  trial  was  2.87  nmi,  with  a  standard  deviation  of  1.05  nmi.  The  minimum  weighted 
slant  range  separation  was  0.66  nmi  and  the  maximum  was  7.74  nmi.  Horizontal  separation  distances 
were  similar,  with  an  average  minimum  horizontal  separation  distance  of  2.69  nmi  and  a  standard 
deviation  of  1.17  nmi  for  all  trials.  iFigure  9provides  a  histogram  of  minimum  weighted  slant  range 
binned  by  1-nmi  increments  up  to  10  nmi,  with  subdivisions  for  0.1  and  0.5  nmi. 
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Figure  9:  Minimum  weighted  slant  range  separation 

The  analysis  also  looked  at  differences  in  minimum  separation  distances  for  each  trial  type.  Minimum 
slant  range  separation  distances  for  envelope  and  risk  ratio  tests  were  similar,  averaging  2.89  nmi  (a: 
1.04  nmi)  and  3.06  nmi  (a:  1.02  nmi),  respectively.  Minima  for  these  test  types  were  also  similar  (0.75 
nmi,  0.76  nmi  respectively)  but  the  maximum  value  for  the  closest  intruder  for  envelope  trials  was  lower 
than  that  for  risk  ratio  trials  (7.49  nmi,  7.74  nmi  respectively).  This  similarity  also  applied  for  horizontal 
separation  distances  (2.73NM,  2.90  nmi  respectively)  and  standard  deviations  (1.11  nmi,  1.10  nmi 
respectively). 

Slant  range  separation  distances  for  the  use  case  trials  were  lower  than  those  for  the  envelope  and  risk 
ratio  trials,  averaging  2.01  nmi  with  a  0.79  nmi  standard  deviation  and  a  range  of  0.66  nmi  to  3.88  nmi. 
These  results  are  shown  in  Figure  10. 


envelope 
risk  ratio 
use  case 


Figure  10:  Minimum  weighted  slant  range  separation  by  trial  type 
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Since  the  use  case  trials  were  designed  for  a  different  purpose  and  in  a  different  manner  than  the 
envelope  and  risk  ratio  tests,  it  is  not  surprising  that  the  average  separation  distances  differ.  When  the 
use  case  trials  are  separated  by  traffic  density,  the  average  separation  distance  for  the  low  density  use 
cases  is  greater  than  those  for  the  medium  and  high  density  use  cases,  as  shown  in  Figure  11.  The 
medium  and  high  density  cases  have  similar  average  slant  range  separation  distances  (1.08  nmi,  1.05 
nmi  respectively)  and  standard  deviations  (0.49  nmi,  0.47  nmi  respectively). 
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Figure  11:  Minimum  slant  range  separation  for  use  case  trials  by  traffic  density  level 

Of  the  112  use  case  trials,  32  had  low  traffic  density:  two  to  four  intruders,  with  an  average  of  three 
intruders  per  trial.  The  remaining  80  trials  were  evenly  split  between  medium  and  high  traffic  density. 
The  medium  density  trials  had  eight  to  10  intruders  and  averaged  8.8  intruders  per  trial,  while  high 
density  trials  each  had  16  to  20  intruders  and  averaged  18  intruders  per  trial. 

Figure  12  shows  the  minimum  horizontal  and  vertical  separation  distances  for  a  subset  of  the  OE  trials. 
Also  shown  on  this  figure  are  the  horizontal  and  vertical  boundaries  for  NMAC  and  self-separation 
violation  parameters.  From  this  figure  it  can  be  seen  that  many  of  the  self-separation  violations 
occurred  on  the  edge  of  the  defined  self-separation  boundary;  similarly,  the  violations  that  were  within 
the  self-separation  bounds  did  not  approach  the  bounds  defined  for  an  NMAC. 
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Figure  12:  Horizontal  and  vertical  separation  distances  for  all  trial  types,  2NM  and  1500'  range,  with  GBSAA  operator  input 

In  comparison,  Figure  13  shows  the  horizontal  and  vertical  minimum  separation  distances  for  the  same 
scenarios  when  no  GBO  input  was  provided.  These  unmitigated  trials  resulted  in  a  total  of  1672  self¬ 
separation  violations  and  56  NMACs.  Note  that  when  GBSAA  is  not  used  each  of  the  eight  runs  of  each 
scenario  would  result  in  an  identical  plot  in  Figure  13.  To  make  these  eight  identical  points  visible  as 
eight  distinct  points  in  the  figure  to  aid  in  comparison  to  Figure  12,  they  were  artificially  spread  out 
slightly  by  adding  a  small  amount  of  noise. 
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Figure  13:  Horizontal  and  vertical  separation  distances  for  all  trial  types,  2  nmi  and  1500'  range,  without  GBSAA  operator 

input 


5.1.5  Subjective  Operator  Responses 

In  addition  to  collecting  quantitative  data  on  operator  performance,  the  OE  evaluation  also  collected 
survey  data  regarding  operator  workload,  performance,  and  factors  that  motivated  maneuver 
recommendation.  Both  workload  and  situation  awareness  were  measured  on  a  five-point  scale,  where 
1  was  equivalent  to  very  low  and  5  was  equivalent  to  very  high.  Overall,  operator  workload  was  very 
low  (mean:  1.62;  a:  0.92),  even  when  the  most  difficult  scenarios  were  encountered,  and  situation 
awareness  was  very  high  (mean:  4.71;  a:  0.75). 
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Figure  14:  Operator  workload,  all  test  types 
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Figure  15:  Operator  situation  awareness,  all  test  types 

Participants  also  found  it  easy  to  use  the  displays  to  perform  the  GBO  tasks.  The  ease  of  use  questions 
were  also  rated  on  a  five-point  scale;  this  time,  a  1  was  equivalent  to  very  easy  and  a  5  was  equivalent  to 
very  difficult.  Operators  thought  it  was  slightly  easier  to  perform  general  GBO  tasks  with  the  Threat 
Display  (mean  1.21;  o0.51)  than  the  Alert  Display  (mean:  1.29;  a:  0.62);  when  asked  about  identifying 
threat  traffic,  the  opposite  was  true  (Threat  Display:  mean  1.33,  o0.56;  Alert  Display:  mean  1.29,  a0.62). 
However,  these  differences  are  not  significant. 

5.1.6  Key  AAR  Qualitative  Insights  and  Observations 

At  the  end  of  each  weekly  session  an  open  discussion  was  held  among  the  test  stakeholders  and  GBOs. 
This  after-action  review  was  designed  to  gather  qualitative  feedback  from  the  operators  based  on  their 
experience  with  the  system.  During  these  sessions,  operators  provided  comments  on  display 
improvements  and  enhancements  and  evaluation  feedback.  The  observations  provided  in  this  section 
were  generally  observed  by  more  than  one  GBO. 

The  GBOs  indicated  that  they  had  received  ample  training  time  and  adequate  materials  to  perform 
successfully  on  the  GBSAA  system.  Overall,  hands-on  familiarization  time  was  reported  as  the  most 
beneficial  training  event. 

The  information  and  its  presentation  on  the  displays  were  positively  received  by  the  GBOs,  who  found 
the  system  easy  to  use.  However,  a  few  suggestions  were  made.  Many  operators  suggested  that  course 
information  be  included  in  the  traffic  data  block.  Similarly,  they  wanted  access  to  more  information 
about  non-alerting  traffic.  Certain  information  that  was  provided  about  threat  traffic,  such  as  vertical 
trend  information,  was  less  visible  for  non-alerting  traffic.  Though  the  datablocks  expanded  when 
operators  clicked  on  them,  this  feature  was  not  apparently  obvious;  additionally,  the  operators  thought 
that  some  of  the  information  shown  in  the  expanded  datablock  should  be  available  in  the  smaller 
datablock. 

Some  operators  expressed  interest  in  consolidating  the  information  onto  a  single  display  to  reduce  the 
division  of  attention,  and  many  operators  commented  that  rapid  changes  to  intruder  prioritization  made 
it  very  difficult  for  them  to  process  the  information  provided  in  the  prioritization  list.  One  operator 
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stated  that  he  would  start  to  provide  a  traffic  call-out  but  before  he  could  complete  it,  the  prioritization 
would  change  and  another  intruder  became  the  priority  intruder. 

As  RFR  weeks  progressed,  GBOs  became  more  familiar  with  the  GBSAA  displays.  Initially,  GBOs  reported 
that  they  most  frequently  relied  on  the  traffic  display  for  situational  awareness.  As  they  became  more 
familiar  with  the  system  they  incorporated  information  presented  on  the  alert  display  into  their  scan 
pattern;  one  operator  commented  that  she  originally  eschewed  using  the  alert  display  since  she  was 
unfamiliar  with  it,  but  as  the  testing  progressed,  she  realized  its  value  to  her  scan. 

Operators  also  provided  feedback  on  the  evaluation  design.  They  noted  that  some  of  the  scenarios 
were  less  predictable  than  what  might  be  seen  in  an  operational  context,  stating  that  some  of  them  felt 
"unwinnable".  They  began  to  anticipate  unexpected  maneuvers  from  intruder  traffic  and  responded 
more  cautiously  as  the  evaluation  progressed.  Operators  also  commented  that  in  a  realistic  use  case, 
they  would  have  been  included  in  a  mission  brief  and  had  access  to  a  mission  flight  plan.  Having  this 
information  would  have  influenced  their  recommended  maneuvers  and  would  have  allowed  them  to 
prioritize  the  mission  to  a  greater  degree.  Finally,  though  the  test  duration  was  similar  to  the  on-task 
hours  for  real-life  operations,  operators  reported  slight  fatigue  as  a  result  of  the  number  and  challenging 
nature  of  the  encounters  presented  each  day. 

Finally,  all  operators  recommended  that  the  GBSAA  system  should  be  fielded. 


6  Conclusions  and  Future  Work 

Operator  performance  throughout  all  three  test  types  indicated  that  the  GBSAA  system  elements  under 
test  as  part  of  the  OE  evaluation  provide  acceptable  levels  of  performance  in  both  simulated  structured 
and  unstructured  airspace..  Over  the  entire  OE  duration,  the  facts  that  no  NMACs  were  observed  and 
that,  relative  to  the  number  of  encounters  experienced,  19  self-separation  violations  occurred,  with  a 
minimum  slant  range  distance  of  0.66  nmi,  provide  evidence  that  the  GBSAA  system  is  likely  to  be  safe 
and  effective  in  supporting  the  SAA  function  in  a  real-life  deployment.  This  performance  supports  the 
operators'  belief  that  the  GBSAA  system  will  warn  them  of  potential  NMAC  and  self-separation 
violations  with  enough  time  to  respond  and  avoid  such  situations. 

In  addition  to  providing  adequate  self-separation,  the  GBSAA  system  proved  easy  to  learn  and  to  use. 
Operators  stated  that  the  training  time  and  materials  they  received  were  sufficient  for  OE  test  purposes, 
and  their  responses  to  the  survey  questions  showed  that  system  use  did  not  incur  much  workload,  even 
during  the  most  stressing  scenarios.  The  GBOs  also  thought  that  the  system  provided  adequate 
situation  awareness  in  all  of  the  scenarios.  All  operators  believed  that  GBSAA  would  be  well-received  in 
the  UAS  community  and  had  high  opinions  of  the  system  at  the  end  of  the  OE  evaluation. 

The  results  of  the  OE  evaluation  indicate  that  the  GBSAA  system  has  the  potential  to  be  a  sufficient 
replacement  for  the  current  SAA  solution  of  ground  observers  and  chase  planes.  Both  operator 
performance  and  feedback  suggest  that  the  system  is  able  to  meet  the  OE  goals  of  safety,  usability,  and 
usefulness,  and  that  it  is  able  to  operate  within  both  structured  airspace  representative  of  a  deployment 
site  and  within  unstructured,  challenging  airspace. 
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Further  evaluation  of  the  use  of  the  production  GBSAA  system  as  a  replacement  for  the  current  SAA 
capability  will  occur  during  flight  tests  at  Dugway  Proving  Grounds,  Utah,  in  the  winter  of  2015/2016. 
Field  operations  are  currently  planned  to  begin  in  2016  at  five  Army  and  two  Air  Force  sites  after  the 
conclusion  of  these  flight  tests.  Additionally,  development  for  Block  1  of  the  GBSAA  system  will 
continue;  this  next  block  will  supply  operators  with  automated  maneuver  guidance  provided  by  the 
Assess  algorithm. 
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8  Acronyms 

AO 

Aircraft  Operator 

ATC 

Air  Traffic  Control 

CPA 

Closest  Point  of  Approach 

DoD 

Department  of  Defense 

FAA 

Federal  Aviation  Administration 

GBO 

Ground-Based  Operator 

GCS 

Ground  Control  Station 

GBSAA 

Ground-Based  Sense  and  Avoid 

HITL 

Human-in-the-Loop 

KGRK 

Robert  Gray  Army  Airfield,  Fort  Hood,  TX 

kn 

Knots 

LSTAR 

Lightweight  Surveillance  and  Target  Acquisition  Radar 

MIT  LL 

Massachusetts  Institute  of  Technology  Lincoln  Laboratory 

NAS 

National  Airspace  System 

nmi 

Nautical  miles 

NMAC 

Near  Mid-Air  Collision 

OE 

Operational  Effectiveness 

RFR 

Runs  for  Record 

SAA 

Sense  and  Avoid 

TAAZ 

Terminal  Area  Alert  Zone 

TCAS 

Traffic  Alert  and  Collision  Avoidance  System 

UAS 

Unmanned  Aircraft  System 
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